Computer processing of human handwriting is an important and fast growing field of application. A number of computers use handwriting as their input. One example is the Apple-Newton (.TM.) which senses writing on its screen. The user places his pen against the screen surface, and the surface records movement of the pen tip thereon. The movement of the pen tip is converted into a signal indicative of the handwriting that has occurred. This handwriting signal is processed in various ways to obtain recognition information about the handwriting signal.
Handwriting can be recognized using many different techniques which are described for example in U.S. Pat. Nos. 5,365,598, 5,347,295, 5,333,209, and 5,181,255, the disclosures of which are herein incorporated by reference. Of course, other techniques are also known for recognizing handwriting, including neural net techniques.
Apart from recognizing handwritten text, the movement of the writing implement can also be used to perform other tasks. One is signature verification. Signature verification compares a person's signature against a stored signature in a database. The signature is "verified" if it matches the signature in the database according to predetermined criteria. These criteria can include, for example, the look of the signature, the speed and style with which the pen is moved, and the like.
Yet another application is the input of drawings and the editing of existing drawings and images for desktop publishing and other activities requiring graphical input.
All of these techniques have required a tablet or a sensitive screen for writing. The tablet senses the movement of the pen by sensing where it touches the tablet surface. The speed of the pen and other parameters can also be sensed. This requires, however, that the tablet be large enough to allow writing on its surface. The tablet must also have certain characteristics which allow it to follow the pen, such as pressure sensitivity or capacitance sensitivity. Some tablets allow paper to be superimposed over the tablet. The ones that do not, however, provide an unnatural feel to the writing since most users are accustomed to obtaining feedback during their writing--they see and feel the writing as it occurs on the paper. In addition, the position of the writing on the screen causes a parallax error, since users are not accustomed to seeing the writing occurring so far from the pen tip. These parameters contribute to a lack of proper feedback which can distort the user's writing enough to make handwriting recognition or verification more difficult.
Another problem, moreover, is that the pens intended to write on the tablet often do not have a writing tip, and therefore their resistance against the tablet surface is different than that to which the user is accustomed. Also, many such systems require a special pen which uses capacitive coupling and this further changes the feel of the pen and the writing.
Also, even in those systems which allow use of normal pens, and/or a sheet of paper superimposed over the screen, the inventors realized a problem that the size of the unit cannot be reduced below a certain limit. This conflicts with the important goal of miniaturization. Computers, and especially notebook computers, are becoming smaller and smaller. Screens are currently available or coming into production which are comparable in size to eyeglass lenses, e.g., one inch square. These screens are viewed through a magnifying glass. The information input device then becomes one factor limiting the size of the computer. The inventors realized that a keyboard cannot be made much smaller than the usual keyboard size without affecting the user's typing. Pen based tablet techniques, moreover, are limited by the size of the special tablet. The user must have sufficient area to write to allow that writing to be recognized, and to prevent the user's writing from being distorted by the size of the tablet.
The special tablet is also often fragile because of its pressure sensitivity.
It is an object of the present invention to provide a graphical input system which does not require a special writing pad and hence has no limits or constraints from the size of the writing pad. A camera and associated processing circuitry is used for graphical input according to the present invention. This different approach allows flexibility. Even though some cameras are just as large as a tablet, others can be made very small.
A further deficiency with the tablet-based systems is that handwriting can only be sensed on the special surface.
Another object of the present invention is to allow monitoring of handwriting on any surface. This allows, for example, handwriting on a whiteboard or chalkboard to be monitored and stored or processed. A printer can be attached to the system to effectively allow forming a printing whiteboard without any special hardware being attached to the whiteboard.
These and other objects are effected according to the present invention by providing a camera which monitors movement of a writing implement relative to a writing surface and associated processing hardware which processes the output of the camera to track that movement.
Some systems which recognize already-written characters are known, including systems like the "digital desk-top", made by Rank/Xerox. These systems do not monitor the movement leading to the writing, but instead monitor the already written words.
This is described in further detail herein.
The present invention teaches a technique of recognizing writing using any writing implement, e.g., a pen, on any writing surface, e.g., a normal piece of paper. The position of the writing implement is preferably tracked by a camera and associated processing equipment. Such cameras are already susceptible of microminiaturization. The camera is placed in a location where it can view and form images of the movement of the pen on the paper. This movement is translated into signals, and these signals are further processed, e.g., recognized as handwriting or converted to an image file.
In one preferred embodiment the initial position of the pen tip is determined. Once the pen tip is determined, the image neighborhood around the initial position of the pen tip is assembled into a "kernel". This kernel is used to determine the position of the pen in subsequent frames. The kernel or image of the tip of the pen is used to scan over the entire image until obtaining a good match. The matching includes moving the kernel on the image, obtaining the product of the image pixel, and summing over all the products. The largest value is the best match.